Principal component analysis as an exploratory data analysis tool

Ryan Miller

Introduction

In his 1977 book Exploratory Data Analysis, John Tukey argued that too much emphasis in statistics was placed on statistical testing (confirmatory analysis), and not enough was placed on exploring the data in search of hypotheses worthy of testing (exploratory analysis)

Introduction

It may have taken a while for Tukey’s views on data exploration to become widely adopted, but they now constitute a major component of data science:

Exploratory Data Analysis

Below are Likert responses (1 = disagree, 5 = agree) to two questions from a large (\(n = 19719\)) online personality survey:

Do you see any trends in the responses?

Exploratory Data Analysis

This personality survey included 50 statements, could you use a graphical approach to understand the trends in responses to all of these statements?

Exploratory Data Analysis

This personality survey included 50 statements, could you use a graphical approach to understand the trends in responses to all of these statements?
- Probably not, that would involve over 1,000 different scatterplots
- Our goal in this talk is to learn about principal component analysis (PCA), an exploratory data analysis tool that is particularly useful for data containing too many variables to explore graphically

Outline

Conceptual Introduction
- Dimension reduction
- What are principal components?
Mathematical Details
- How are principal components derived?
- How many principal components are necessary?
Applications
- Big 5 personality traits
- Minnesota colleges
- FIFA 2018

Dimensions

Consider a relatively simple dataset documenting the murder and assault rates (per 100k population) of the 50 US states
- Each observation (state) exists in two-dimensions and is defined by coordinates on the “Murder” and “Assault” axes

Do both of these dimensions appear to provide unique information?

Dimension Reduction

Notice how these two variables are correlated, meaning they could be representing the same underlying factor (which we might describe as “violent crime”)
- If we mapped a state’s coordinates (in terms of the original two variables) into a score in this underlying factor we could represent these data using just a single dimension

Dimension Reduction

Here are what the “violent crime” scores look like for the 50 states:

This is an example dimension reduction, we took two-dimensional data (murder and assault rates) and reduced it a single dimension (“violent crime” scores)

Three-Dimensions

The fundamental concept in principal component analysis (PCA) is that data with many variables can often be effectively represented using smaller set of derived dimensions
In the example below, one-dimension (a line) provides an effective representation of the original three-dimensional data:

In this second example, a line provides a poor representation, but a plane (two dimensions) can provide a good representation:

What are Principal Components?

These derived dimensions are called principal components, and they are constructed in a very particular way:
- They’re sequential, the first principal component (PC1) describes the direction of maximal variation
- They’re orthogonal, all principal components are independent of each other
There are always the same number of principal components as there are variables, but often these components are not equally interesting

In panel A, PC1 contains most of the variation in the data (indicating a one-dimensional representation might be effective)
In panel B, PC1 and PC2 both contain substantial variation (indicating a one-dimensional representation would lose substantial information)

Standardization

How “variation” is quantified (think standard deviation/variance) depends upon the units of variables involved
This can have undesirable consequences when looking for the direction of maximal variation

Standardization

Standardization (representing your variables using z-scores) addresses this concern:

\[z_{ij} = \tfrac{x_{ij} - \bar{x}_j}{s_{j}}\]

In applications where the original variables have different units (scales), standardization is particularly important

What are Principal Components?

Each principal component is a single dimension (ie: a line)
Each observation’s score (position) on this line is given by a linear combination of the original variables: \[Z = \phi_{1} X_1 + \phi_{2} X_2 + \ldots + \phi_{p} X_p\]
The coefficients \(\phi_1, \ldots, \phi_p\) are sometimes called loadings
In the US Arrests example, here is the first principal component: \[Z_1 = 0.71\text{Murder} + 0.71\text{Assault} \]

Mathematical Details - Covariance Matrix

To find the loadings we’ll need to consider the matrix \(\tfrac{1}{n-1}\mathbf{X}^T\mathbf{X}\)
This is known as the covariance matrix because it describes the relationship between pairs of variables
- If the data have been standardized this matrix is known as the correlation matrix

X <- scale(X)
1/(nrow(X) - 1) * t(X) %*% X  ## Calculating it ourselves

##            Murder   Assault
## Murder  1.0000000 0.8018733
## Assault 0.8018733 1.0000000

Mathematical Details - Decomposition

The covariance matrix, \(\mathbf{C} = \tfrac{1}{n-1}\mathbf{X}^T\mathbf{X}\), can be decomposed using an eigenvalue decomposition, a technique from linear algebra: \[\mathbf{C} = \mathbf{V}\mathbf{L}\mathbf{V}^T\]
\(\mathbf{V}\) is a matrix of eigenvectors, while \(\mathbf{L}\) is a diagonal matrix of eigenvalues
The columns of \(\mathbf{V}\) contain the loadings of each principal component
The elements of \(\mathbf{L}\) describe the variation that exists in each principal component

C <- 1/(nrow(X) - 1) * t(X) %*% X 
eigen(C)

## eigen() decomposition
## $values
## [1] 1.8018733 0.1981267
## 
## $vectors
##           [,1]       [,2]
## [1,] 0.7071068 -0.7071068
## [2,] 0.7071068  0.7071068

Principal Component Analysis (PCA)

When using PCA as an exploratory data analysis tool the mathematical details are less important than the conceptual ones
- PCA helps represent many related variables using a smaller set of underlying factors (principal components)

Application #1 - The Big Five

Human personalities are complex, but can they be effectively boiled down to a small set of underlying dimensions?
Earlier we introduced data from a large (\(n = 19719\)) personality survey where participants rated their agreement with 50 different statements:

## E1   I am the life of the party.
## E2   I don't talk a lot.
## E3   I feel comfortable around people.
## E4   I keep in the background.
## E5   I start conversations.
## E6   I have little to say.
## E7   I talk to a lot of different people at parties.
## E8   I don't like to draw attention to myself.
## E9   I don't mind being the center of attention.
## E10  I am quiet around strangers.
## N1   I get stressed out easily.
## N2   I am relaxed most of the time.
## N3   I worry about things.
## N4   I seldom feel blue.
## N5   I am easily disturbed.
## N6   I get upset easily.
## N7   I change my mood a lot.
## N8   I have frequent mood swings.
## N9   I get irritated easily.
## N10  I often feel blue.
## A1   I feel little concern for others.
## A2   I am interested in people.
## A3   I insult people.
## A4   I sympathize with others' feelings.
## A5   I am not interested in other people's problems.
## A6   I have a soft heart.
## A7   I am not really interested in others.
## A8   I take time out for others.
## A9   I feel others' emotions.
## A10  I make people feel at ease.
## C1   I am always prepared.
## C2   I leave my belongings around.
## C3   I pay attention to details.
## C4   I make a mess of things.
## C5   I get chores done right away.
## C6   I often forget to put things back in their proper place.
## C7   I like order.
## C8   I shirk my duties.
## C9   I follow a schedule.
## C10  I am exacting in my work.
## O1   I have a rich vocabulary.
## O2   I have difficulty understanding abstract ideas.
## O3   I have a vivid imagination.
## O4   I am not interested in abstract ideas.
## O5   I have excellent ideas.
## O6   I do not have a good imagination.
## O7   I am quick to understand things.
## O8   I use difficult words.
## O9   I spend time reflecting on things.
## O10  I am full of ideas.

Application #1 - The Big Five

We expect these ratings are correlated, meaning there might be a smaller set of hidden factors that describe a person’s responses

cov(bigFive)[1:5,1:5]

##            E1         E2         E3         E4         E5
## E1  1.0000000 -0.4210091  0.4742752 -0.4837811  0.4791013
## E2 -0.4210091  1.0000000 -0.4455374  0.5277232 -0.5395176
## E3  0.4742752 -0.4455374  1.0000000 -0.4810062  0.5906601
## E4 -0.4837811  0.5277232 -0.4810062  1.0000000 -0.5100685
## E5  0.4791013 -0.5395176  0.5906601 -0.5100685  1.0000000

Application #1 - The Big Five

We’ve now got an indication that PCA might be useful in helping us find a set of hidden factors that represent how subjects rate these statements
But how many of these factors are there?

Application #1 - The Big Five

We’ve now got an indication that PCA might be useful in helping us find a set of hidden factors that represent how subjects rate these statements
But how many of these factors are there?
- The answer somewhat subjective, but the guiding principle is that each factor should be informative
- Informativeness can be measured by the variance that exists in a component, or its “variance explained”

Application #1 - The Big Five

A scree plot is a popular method for visualizing the variance explained by the first \(K\) principal components

Five principal component dimensions can represent nearly 50% of the variation in individual responses to these 50 statements!

Application #1 - The Big Five

Looking at the statements that are the largest contributors to PC1, most of them are questions from the “E-series”
This is not a coincidence, these questions were categorized as targeting extroversion vs. introversion, the first of the Big Five traits

## E3   I feel comfortable around people.
## E4   I keep in the background.
## E5   I start conversations.
## E7   I talk to a lot of different people at parties.
## E10  I am quiet around strangers.
## N10  I often feel blue.

Application #1 - The Big Five

Below are the top contributors to PC2 - PC5
Notice how the Big Five traits: extroversion, neuroticism, conscientiousness, openness to experiences, and agreeableness clearly emerge!

This example illustrates how a larger set of variables can be understood as representing a smaller set of underlying factors
- PCA provides a tool for identifying these factors

Application #2 - Minnesota Colleges

The College Scorecard is a government run tool that allows users to compare US Colleges
It uses a publicly available dataset containing hundreds of variables documenting every college in the United States
- But are hundreds of variables really necessary to compare colleges? Could they be represented by a smaller number of underlying factors?

Application #2 - Minnesota Colleges

Shown below is the covariance matrix of 9 variables from the College Scorecard data
- Do you think dimension reduction is possible?

##               admissionRate ACTmath ACTenglish undergrads  cost gradRate
## admissionRate          1.00   -0.44      -0.47      -0.09 -0.41    -0.38
## ACTmath               -0.44    1.00       0.94       0.13  0.58     0.90
## ACTenglish            -0.47    0.94       1.00       0.04  0.65     0.89
## undergrads            -0.09    0.13       0.04       1.00 -0.38    -0.04
## cost                  -0.41    0.58       0.65      -0.38  1.00     0.73
## gradRate              -0.38    0.90       0.89      -0.04  0.73     1.00
## FYretention           -0.41    0.91       0.88       0.03  0.69     0.94
## fedloan                0.23   -0.23      -0.12      -0.31  0.25     0.00
## debt                   0.12   -0.10      -0.03      -0.20  0.38     0.12
##               FYretention fedloan  debt
## admissionRate       -0.41    0.23  0.12
## ACTmath              0.91   -0.23 -0.10
## ACTenglish           0.88   -0.12 -0.03
## undergrads           0.03   -0.31 -0.20
## cost                 0.69    0.25  0.38
## gradRate             0.94    0.00  0.12
## FYretention          1.00   -0.05  0.08
## fedloan             -0.05    1.00  0.55
## debt                 0.08    0.55  1.00

Application #2 - Minnesota Colleges

A biplot is popular method for visualizing the loadings and scores for pair of principal components
Shown below is the biplot resulting from PCA performed on 9 variables describing 32 colleges in Minnesota

The x-axis and y-axis of this plot describe each college’s score in the first two principal components
The arrows describe each variable’s loading on these two dimensions

Application #2 - Minnesota Colleges

PC1 appears to represent “selectiveness”
- Schools with above average ACT scores, retention, and graduation rates have low scores in this dimension
PC2 appears to represent a contrast between “size” and “financial burden”
- Schools with above average undergrad enrollments, and/or below average debt and fed loan rates, have high scores in this dimension

Application #3 - FIFA 2018

FIFA is a popular series of soccer (football) video games
The performance of players in FIFA is determined by their ratings in 33 different attributes
The website sofifa.com documents these attributes
- But are all of these attributes really necessary to understanding FIFA players?

	crossing	finishing	heading_accuracy	short_passing	volleys	dribbling	curve	free_kick_accuracy	long_passing	ball_control	acceleration	sprint_speed	agility	reactions	balance	shot_power	jumping	stamina	strength	long_shots	aggression	interceptions	positioning	vision	penalties	composure	marking	standing_tackle	sliding_tackle	gk_diving	gk_handling	gk_kicking	gk_positioning	gk_reflexes
Cristiano Ronaldo	85	94	88	83	88	91	81	76	77	93	89	91	89	96	63	94	95	92	80	92	63	29	95	85	85	95	22	31	23	7	11	15	14	11
L. Messi	77	95	71	88	85	97	89	90	87	95	92	87	90	95	95	85	68	73	59	88	48	22	93	90	78	96	13	28	26	6	11	15	14	8
Neymar	75	89	62	81	83	96	81	84	75	95	94	90	96	88	82	80	61	78	53	77	56	36	90	80	81	92	21	24	33	9	9	15	15	11
L. SuÃ¡rez	77	94	77	83	88	86	86	84	64	91	88	77	86	93	60	87	69	89	80	86	78	41	92	84	85	83	30	45	38	27	25	31	33	37
M. Neuer	15	13	25	55	11	30	14	11	59	48	58	61	52	85	35	25	78	44	83	16	29	30	12	70	47	70	10	10	11	91	90	95	91	89
R. Lewandowski	62	91	85	83	87	85	77	84	65	89	79	83	78	91	80	88	84	79	84	83	80	39	91	78	84	87	25	42	19	15	6	12	8	10
De Gea	17	13	21	50	13	18	21	19	51	42	57	58	60	88	43	31	67	40	64	12	38	30	12	68	40	64	13	21	13	90	85	87	86	90
E. Hazard	80	83	57	86	79	93	82	79	81	92	93	87	93	85	91	79	59	79	65	82	54	41	85	86	86	87	25	27	22	11	12	6	8	8
T. Kroos	85	76	54	90	82	79	85	84	93	89	60	52	71	86	69	87	32	77	74	90	60	85	79	88	73	85	63	82	69	10	11	13	7	10
G. HiguaÃn	68	91	86	75	88	84	74	62	59	85	78	80	75	88	69	88	79	72	85	82	50	20	92	70	70	86	12	22	18	5	12	7	5	10
Sergio Ramos	66	60	91	78	66	61	73	67	72	84	75	77	79	85	60	79	93	84	81	55	84	88	52	63	68	80	86	89	91	11	8	9	7	11
K. De Bruyne	90	83	53	90	82	85	83	83	84	87	76	75	80	88	75	85	65	87	73	86	68	56	84	90	77	84	30	51	40	15	13	5	10	13
T. Courtois	14	14	13	32	12	13	19	11	31	23	46	52	61	81	45	36	68	38	70	17	23	15	13	44	27	52	11	18	16	85	91	69	86	88
A. SÃ¡nchez	80	85	70	81	83	90	78	78	73	87	88	84	90	87	87	84	85	85	72	82	80	42	86	83	77	86	30	39	35	10	10	15	12	13
L. ModriÄ	78	71	55	92	74	86	79	77	83	92	75	71	93	88	94	73	67	82	58	82	62	76	79	90	80	84	66	80	73	13	9	7	14	9
G. Bale	87	87	86	85	76	87	86	85	80	86	93	95	77	87	65	91	85	76	80	90	65	59	86	79	76	85	51	55	52	15	15	11	5	6
S. AgÃ¼ero	70	90	68	79	85	89	82	72	63	89	90	84	86	89	91	88	80	74	74	83	63	24	91	83	83	90	13	20	12	13	15	6	11	14
G. Chiellini	58	33	84	59	45	58	60	31	59	57	68	78	59	82	64	78	89	68	91	49	92	88	28	50	50	82	92	92	90	3	3	2	4	3
G. Buffon	13	15	13	37	17	26	20	13	35	28	49	43	55	80	49	39	75	39	69	13	38	28	12	50	22	70	10	11	11	89	88	74	90	84
P. Dybala	80	85	68	83	88	92	88	84	71	93	88	84	91	84	85	82	75	83	65	88	48	24	84	84	86	84	14	20	20	5	4	4	5	8
J. Oblak	13	11	15	29	13	12	13	14	26	16	43	60	67	84	49	22	76	41	78	12	34	19	11	55	11	55	14	12	18	84	90	77	87	84
A. Griezmann	82	88	80	79	87	87	84	75	75	86	87	86	90	90	80	81	87	76	61	82	69	35	91	76	71	86	23	22	11	14	8	14	13	14
Thiago	72	69	58	91	90	90	85	77	90	92	77	68	90	84	86	75	75	75	59	82	57	78	83	86	75	83	49	64	62	6	11	7	9	13
P. Aubameyang	77	88	79	77	86	82	78	74	64	82	95	96	77	87	70	81	77	79	77	79	43	48	89	77	74	84	28	25	36	6	9	15	9	9
L. Bonucci	44	39	88	79	58	69	56	61	85	75	62	72	60	86	52	74	85	73	85	65	82	90	38	74	70	84	84	88	88	2	2	3	2	4
J. Boateng	69	34	85	75	53	67	56	31	80	71	72	78	58	82	53	79	75	74	91	58	82	83	47	76	46	86	88	91	90	7	12	15	6	5
D. GodÃn	55	42	92	79	47	53	49	51	70	76	64	70	63	85	58	67	89	67	80	43	88	88	48	52	50	82	87	86	89	6	8	15	5	15
M. Hummels	64	55	89	80	60	68	65	53	85	77	62	65	64	85	58	71	68	66	85	51	66	89	56	79	68	91	85	92	90	15	6	10	5	6
M. Ãzil	82	74	54	89	77	86	84	79	84	90	75	68	83	87	70	70	52	69	57	76	48	24	83	92	67	78	22	16	19	6	14	10	6	14
H. Lloris	13	10	10	50	11	10	11	10	50	34	65	63	55	85	54	23	74	41	43	14	31	27	10	30	40	61	12	10	18	88	86	68	82	90
Thiago Silva	60	38	82	79	63	68	61	67	81	80	70	74	74	84	68	74	93	74	81	71	77	89	59	74	71	83	90	89	88	9	12	5	9	10
Z. IbrahimoviÄ	76	87	80	84	90	82	82	82	76	88	63	67	81	86	41	91	69	71	88	85	84	20	86	83	91	91	15	41	27	13	15	10	9	12
A. Robben	80	85	51	84	86	92	87	81	71	89	87	86	89	87	91	87	61	68	67	88	47	39	85	83	80	86	29	26	26	10	8	11	5	15
N. KantÃ©	67	65	54	84	56	77	49	49	79	79	82	80	83	87	90	72	79	94	77	66	90	91	71	76	54	84	75	89	85	15	12	10	7	10
M. Verratti	72	58	55	90	64	90	73	64	89	88	69	62	87	87	94	56	64	77	56	58	89	86	71	88	64	91	76	84	79	12	12	15	15	10
P. Pogba	78	71	73	86	85	86	84	82	88	89	72	76	76	84	61	90	85	91	91	82	75	68	83	86	76	85	66	68	66	5	6	2	4	3
C. Eriksen	86	76	52	89	74	84	85	87	86	87	77	74	78	83	82	80	57	90	57	84	44	55	81	90	67	87	38	56	22	9	14	7	7	6
A. Vidal	76	77	81	84	78	76	76	68	82	83	76	74	75	89	77	86	82	91	77	85	93	85	80	80	81	86	76	89	85	4	2	4	2	4
E. Cavani	70	83	88	78	84	80	75	74	50	80	75	78	81	90	59	85	88	92	74	78	84	48	92	77	84	83	24	45	27	12	5	13	13	10
Marcelo	90	67	71	82	54	86	80	67	76	88	77	79	77	87	82	83	77	81	77	66	84	86	75	75	59	85	80	85	87	12	5	5	5	9
M. HamÅ¡Ãk	78	78	74	88	79	86	80	72	82	87	72	75	76	88	66	82	71	85	69	83	56	72	88	86	70	84	60	73	62	8	6	4	14	14
I. RakitiÄ	81	83	58	87	80	84	88	84	92	87	67	64	72	79	66	84	44	76	63	88	67	69	80	87	78	81	52	65	53	14	11	12	5	9
David Silva	83	71	54	91	79	84	82	75	79	90	72	65	92	84	89	71	66	78	56	73	51	43	81	92	73	92	25	44	29	1	1	1	1	1
S. HandanoviÄ	12	10	10	36	12	18	12	14	34	24	54	57	42	83	36	22	78	41	71	19	25	22	12	41	23	62	17	10	13	87	86	69	87	87
PiquÃ©	57	64	83	81	57	59	58	43	80	78	55	73	58	84	42	62	74	66	83	51	72	88	58	62	69	86	86	87	86	10	11	14	15	8
Iniesta	77	70	54	92	74	90	80	70	86	94	72	71	79	88	84	65	52	58	58	71	58	66	84	94	71	89	57	57	56	6	13	6	13	7
H. Kane	70	90	83	75	73	78	71	66	67	82	70	75	73	87	62	87	70	85	86	84	74	35	89	75	86	84	41	36	38	8	10	11	14	11
J. RodrÃguez	88	82	62	84	91	83	87	86	80	85	77	75	82	84	75	86	54	68	70	91	65	55	80	86	81	87	25	42	30	15	15	15	5	14
Isco	72	88	55	86	64	90	86	74	85	91	74	69	83	79	85	72	64	71	59	87	58	70	83	84	76	86	25	73	51	10	8	12	15	6
D. Alaba	82	63	75	83	68	79	78	85	80	83	86	86	79	84	77	83	82	87	69	83	69	85	77	78	80	79	81	83	83	5	7	14	15	9
R. Lukaku	68	89	87	71	76	78	72	66	68	75	76	89	62	85	47	88	71	80	93	75	72	27	87	73	82	84	27	30	30	8	15	14	7	10
Alex Sandro	86	55	76	81	68	82	78	59	71	81	84	88	79	86	79	80	79	93	80	72	82	82	70	70	60	82	78	84	84	7	7	9	12	5
T. MÃ¼ller	77	87	82	81	83	75	81	59	68	82	76	78	76	92	72	77	81	90	67	77	57	58	93	81	60	83	31	41	44	6	7	11	14	14
Sergio Busquets	62	60	68	88	44	74	66	68	79	80	48	34	58	82	55	61	66	88	77	54	85	86	77	84	60	88	81	85	80	5	8	13	9	13
Coutinho	77	74	47	85	75	88	84	82	78	88	88	77	92	81	92	80	59	80	61	86	55	30	83	87	64	85	24	44	40	12	7	9	14	6
M. Reus	80	86	49	86	88	86	90	84	80	85	86	90	86	87	81	87	72	68	67	88	49	50	86	86	87	83	30	36	46	12	12	13	13	11
T. Alderweireld	64	45	82	77	38	62	63	69	81	70	62	69	60	86	50	86	81	76	82	66	81	84	56	67	60	81	89	90	85	16	6	14	16	14
David Luiz	68	55	83	79	64	66	70	76	78	79	68	71	72	79	56	77	80	76	81	71	86	87	54	69	75	78	80	85	84	11	12	10	7	14
Diego Costa	65	88	83	67	81	77	62	59	52	83	74	75	58	86	52	84	64	83	91	73	93	40	88	74	76	84	28	39	34	11	13	12	8	11
R. Nainggolan	73	76	59	84	75	80	73	68	81	85	80	76	81	87	84	84	76	94	76	86	88	86	88	76	63	85	78	86	88	11	11	14	8	11
Javi MartÃnez	60	53	90	75	61	62	54	45	74	71	49	50	54	84	58	76	83	64	84	63	91	86	57	74	52	83	84	87	87	5	12	9	11	8
D. Mertens	77	85	35	81	70	89	81	77	74	88	93	87	94	85	92	80	60	75	42	81	59	36	85	82	78	81	30	40	40	7	10	8	10	4
Sokratis	47	44	87	64	46	58	36	35	64	58	71	84	62	81	68	79	87	74	86	48	91	80	50	52	33	82	88	90	86	6	8	5	12	5
Miranda	48	43	83	70	51	65	32	39	69	72	67	74	61	82	57	70	82	65	80	41	83	88	43	58	49	84	89	90	86	12	6	10	13	12
K. Benzema	75	84	78	86	79	82	79	73	55	86	77	78	77	83	60	83	70	73	78	77	65	22	87	85	82	79	13	15	12	13	11	5	5	7
Cesc FÃ bregas	85	76	74	91	81	80	77	81	90	86	65	62	65	81	77	77	68	80	64	75	45	50	78	91	80	92	60	63	62	6	10	8	15	15
F. RibÃ©ry	81	76	41	86	80	90	83	81	75	90	85	79	88	86	91	76	34	64	60	75	52	36	81	86	78	86	13	25	26	15	6	9	7	10
Pepe	46	46	80	70	23	58	44	47	61	62	65	70	63	83	49	63	76	62	84	56	94	86	40	48	57	77	86	86	87	8	15	5	9	10
P. Äech	19	12	19	35	17	12	13	19	33	22	40	44	49	84	34	21	51	32	65	11	17	23	13	53	23	70	11	13	12	82	87	76	83	81
Y. Carrasco	80	79	53	79	68	91	81	78	74	86	87	89	91	83	71	84	68	73	59	75	61	18	82	74	74	75	19	39	26	9	11	9	10	10
R. Varane	36	40	83	75	42	57	46	54	76	77	80	85	58	81	47	52	80	68	85	53	77	82	44	56	43	82	84	86	85	11	11	9	5	14
Morata	73	84	83	78	81	83	76	44	51	84	79	87	65	81	60	80	79	73	82	76	56	24	85	76	70	80	14	14	20	4	5	4	4	5
Casemiro	52	59	74	82	53	69	59	74	84	78	63	61	66	83	57	86	87	85	85	77	87	86	66	76	66	86	83	87	86	13	14	16	12	12
L. Insigne	83	73	56	83	74	88	83	77	73	90	94	86	96	80	93	70	53	75	44	80	34	26	83	87	61	78	16	24	22	8	4	14	9	10
Koke	84	71	61	86	56	80	85	76	85	86	74	70	77	84	79	84	61	84	75	81	71	66	82	89	59	75	66	69	51	14	12	5	10	13
A. Lacazette	64	87	72	79	82	85	74	71	59	84	84	87	87	85	84	81	76	80	79	76	70	40	87	74	84	86	23	42	30	11	6	9	5	6
K. Navas	11	15	11	30	11	16	11	15	37	19	54	53	60	82	61	21	74	39	75	13	32	20	16	54	25	49	12	14	14	89	78	72	80	88
H. Mkhitaryan	79	77	53	84	81	87	83	64	72	87	88	84	89	82	85	87	71	86	63	75	69	64	82	85	72	79	49	53	55	5	11	14	5	7
D. SubaÅ¡iÄ	11	10	13	24	13	11	24	66	26	19	51	54	42	82	37	24	75	32	80	14	31	20	15	52	23	65	15	14	13	84	79	79	85	87
B. Leno	9	9	13	37	10	16	9	8	33	22	46	52	52	81	44	23	73	43	68	14	28	22	7	53	23	62	8	15	18	85	84	74	85	83
M. ter Stegen	15	14	11	30	14	17	18	12	38	18	38	50	37	82	43	22	79	35	79	10	43	22	11	57	25	62	10	13	10	84	83	87	81	86
K. Glik	29	27	85	63	17	51	35	26	59	59	53	58	34	79	55	57	82	72	90	20	86	87	24	23	39	79	85	87	83	10	14	5	8	10
Jordi Alba	82	73	63	79	60	80	77	64	70	83	93	93	90	82	86	64	81	90	60	66	75	83	77	68	59	75	81	84	83	13	15	13	6	13
I. GÃ¼ndoÄan	74	72	49	88	75	87	81	74	83	88	73	70	83	87	81	77	62	71	74	74	68	79	79	87	76	86	57	67	64	11	14	5	5	13
Azpilicueta	78	46	76	80	51	69	66	58	74	79	78	79	75	86	73	68	76	88	73	60	80	85	62	68	66	78	87	88	87	13	9	10	6	5
A. Di MarÃa	85	74	53	83	77	86	78	72	79	85	86	86	92	80	79	83	72	78	60	77	73	38	82	81	73	82	37	59	56	10	7	11	12	11
M. PjaniÄ	80	61	60	89	72	86	86	91	85	89	74	74	79	84	79	78	61	78	65	82	68	74	70	88	79	84	62	70	66	7	7	13	7	8
C. Marchisio	76	61	68	87	76	84	78	78	85	85	78	78	81	86	79	82	68	81	70	82	78	80	80	84	78	84	72	78	80	4	3	4	2	4
J. Vertonghen	68	56	81	79	52	69	58	73	74	74	68	66	62	82	55	86	83	79	81	68	83	86	60	68	68	81	84	86	86	6	10	9	12	7
B. Matuidi	75	65	73	84	72	73	65	52	79	77	79	78	76	84	79	74	84	94	78	66	85	85	68	75	56	82	83	84	81	8	11	5	10	14
S. Ruffier	11	11	12	33	13	12	17	12	34	15	45	50	51	78	55	21	72	37	76	11	31	25	12	49	25	57	11	12	11	82	84	77	85	87
Filipe LuÃs	84	59	68	78	56	77	74	75	77	81	75	82	74	83	60	71	63	91	74	60	81	82	71	70	65	88	85	83	86	11	15	9	9	5
V. Kompany	61	45	84	72	46	63	61	52	66	68	61	64	60	84	42	75	71	74	88	55	87	80	41	59	63	81	84	85	84	10	9	5	8	6
A. Barzagli	40	21	72	65	35	64	50	31	65	65	64	76	51	80	64	54	85	46	87	39	70	92	28	62	60	82	90	90	88	4	2	4	2	4
E. Bailly	51	33	81	72	42	63	46	43	56	68	75	80	63	78	63	55	84	78	85	48	87	85	47	54	46	74	82	85	85	14	14	10	9	8
Marco Asensio	79	82	50	86	75	83	83	76	81	86	83	82	78	78	76	84	57	73	61	87	45	45	72	83	62	82	29	42	39	13	9	6	12	10
Bernardo Silva	79	76	51	84	66	91	79	63	78	89	85	78	88	84	86	72	53	80	54	64	58	49	77	85	64	84	46	44	35	9	10	14	12	9
A. Laporte	58	45	85	73	40	55	61	69	78	71	68	70	49	82	55	68	77	71	84	47	72	85	54	52	35	74	86	88	85	10	11	5	14	5
D. Alli	68	84	75	83	76	83	71	53	76	85	77	76	74	85	62	77	69	89	71	79	84	67	86	83	68	86	60	63	57	7	6	9	11	8
S. ManÃ©	72	83	53	77	68	86	74	64	71	84	94	92	89	80	85	77	75	82	72	69	73	31	84	78	71	78	33	35	38	10	10	15	7	14

Application #3 - FIFA 2018

Based upon the scree plot, do you think these 33 attributes could be effectively represented by two principal components?

Application #3 - FIFA 2018

Based upon the biplot, how might you describe the first principal component?
- And what is going on in the right side of the plot?

Application #3 - FIFA 2018

The biplot clearly differentiates goalkeepers from other players

Application #3 - FIFA 2018

Nothing forces us to analyze all players at once, below we apply PCA after filtering out goalkeepers

Notice how filtering leads to different loadings and amounts of variance explained by component
- This brings us back to the cylclic nature of exploratory data analysis
- PCA -> identify interesting subpopulation -> PCA -> \(\ldots\)

Conclusion

PCA is a tool for representing data containing many variables using a smaller set of derived dimensions
- These derived dimensions can be interpretted as underlying factors that were measured by the full set of original variables

Conclusion

PCA is a tool for representing data containing many variables using a smaller set of derived dimensions
- These derived dimensions can be interpretted as underlying factors that were measured by the full set of original variables
Ideally we can find a small number of underlying factors that explain most of the variance in the data
- We could determine this using a scree plot

Conclusion

PCA is a tool for representing data containing many variables using a smaller set of derived dimensions
- These derived dimensions can be interpretted as underlying factors that were measured by the full set of original variables
Ideally we can find a small number of underlying factors that explain most of the variance in the data
- We could determine this using a scree plot
- We could then visualize relative locations of data-points using a biplot